Recurrent neural network for tumor movement prediction

ABSTRACT

In an embodiment, there is provided a method of predicting respiratory motion in real-time. The method includes training, by a training module, a recurrent neural network circuitry in real time. The training is performed over a training time interval. The training is based, at least in part, on a training data set. The training data set includes a training number of measured position samples. The measured position samples are related to respiratory motion of a patient target region. The method further includes predicting, by the trained recurrent neural network circuitry, a future position of the patient target region at a future point in time based, at least in part, on a prediction data set. The future point in time is a look ahead time interval in the future relative to a prediction time interval. The prediction data set includes a prediction number of measured position samples.

CROSS REFERENCE TO RELATED APPLICATION(S)

This application claims the benefit of U.S. Provisional Application No.62/983,097, filed Feb. 28, 2020, which is incorporated by reference asif disclosed herein in its entirety.

GOVERNMENT LICENSE RIGHTS

This invention was made with government support under grant numberFA8750-15-3-6000, awarded by the Air Force Research Laboratory. At leasta portion of the invention was made with government support undercontract number SA2018-UP-020-NOA, with the Air Force ResearchLaboratory. The government has certain rights in the invention.

FIELD

The present disclosure relates to tumor movement prediction, inparticular to, a recurrent neural network for tumor movement prediction.

BACKGROUND

Respiratory motion is a well-recognized challenge in chest and upperabdominal imaging (e.g., computerized tomography (CT) scan, magneticresonance imaging (MRI), mammography), and radiation therapy for tumorsincluded in chest or upper abdominal organs (e.g., lung, liver, throat,stomach, breast and/or pancreas). Tumor motion caused by respiration mayfollow a relatively complex moving trajectory driven by a patientbreathing pattern. Patient respiration may vary over time in one sessionor between sessions for the same patient and may differ significantlybetween patients. For example, the patient may cough or sneeze, and/or arespiration pattern may change as the patient experiences pain. Thevariation in patient respiration may include variation in amplitudeand/or frequency and may result in drift in a location of a targetedtumor during a treatment session. Such drift may then result in exposureof healthy tissue to radiation and/or reduced efficiency for thetargeted tumor. Conventional techniques for counteracting respiratorymotion may include breath-hold, passive motion tracking and respirationgating (wait) methods. The conventional techniques may result in patientdiscomfort and/or increased treatment time and thus increased radiationexposure.

SUMMARY

In an embodiment, there is provided a method of predicting respiratorymotion in real-time. The method includes training, by a training module,a recurrent neural network circuitry in real time. The training isperformed over a training time interval. The training is based, at leastin part, on a training data set. The training data set includes atraining number of measured position samples. The measured positionsamples are related to respiratory motion of a patient target region.The method further includes predicting, by the trained recurrent neuralnetwork circuitry, a future position of the patient target region at afuture point in time based, at least in part, on a prediction data set.The future point in time is a look ahead time interval in the futurerelative to a prediction time interval. The prediction data set includesa prediction number of measured position samples.

In some embodiments, the method further includes updating, by thetraining module, the training of the recurrent neural network circuitry,at an update time interval. The updating is based, at least in part, ona current internal state of the recurrent neural network circuitry.

In some embodiments of the method, the training includes repeatedlydetermining a plurality of output weight values using a sliding windowcorresponding to the prediction number of measured position samples.

In some embodiments of the method, the training data set corresponds toa sliding window.

In some embodiments of the method, the prediction time interval is lessthan one second. In some embodiments of the method, the update timeinterval is related to a position data sample rate.

In some embodiments of the method, the recurrent neural networkcorresponds to a reservoir computing network.

In some embodiments of the method, the training time interval is lessthan or equal to 30 seconds. In some embodiments of the method, theprediction number of samples is in the range of 5 to 20. In someembodiments of the method, the position data sample rate is in the rangeof 30 to 40 samples per second.

In an embodiment, there is provided a system for predicting respiratorymotion in real time. The system includes a system controller circuitry,a training module, and a recurrent neural network circuitry. Thetraining module is configured to train the recurrent neural networkcircuitry in real time. The training is performed over a training timeinterval. The training is based, at least in part, on a training dataset. The training data set includes a training number of measuredposition samples. The measured position samples are related torespiratory motion of a patient target region. The trained recurrentneural network circuitry is configured to predict a future position ofthe patient target region at a future point in time based, at least inpart, on a prediction data set. The future point in time is a look aheadtime interval in the future relative to a prediction time interval. Theprediction data set includes a prediction number of measured positionsamples.

In some embodiments of the system, the training module is configured toupdate the training of the recurrent neural network circuitry, at anupdate time interval. The updating is based, at least in part, on acurrent internal state of the recurrent neural network circuitry.

In some embodiments of the system, the training includes repeatedlydetermining a plurality of output weight values using a sliding windowcorresponding to the prediction number of measured position samples.

In some embodiments of the system, the training data set corresponds toa sliding window.

In some embodiments of the system, the prediction time interval is lessthan one second. In some embodiments of the system, the update timeinterval is related to a position data sample rate.

In some embodiments of the system, the recurrent neural networkcorresponds to a reservoir computing network.

In some embodiments of the system, the training time interval is lessthan or equal to 30 seconds.

In some embodiments of the system, the prediction number of samples isin the range of 5 to 20.

In some embodiments of the system, the position data sample rate is inthe range of 30 to 40 samples per second.

BRIEF DESCRIPTION OF DRAWINGS

The drawings show embodiments of the disclosed subject matter for thepurpose of illustrating features and advantages of the disclosed subjectmatter. However, it should be understood that the present application isnot limited to the precise arrangements and instrumentalities shown inthe drawings, wherein:

FIG. 1 illustrates a functional block diagram of a system for tumormovement prediction, consistent with several embodiments of the presentdisclosure; and

FIG. 2 illustrates a functional block diagram of an example recurrentneural network consistent with several embodiments of the presentdisclosure;

FIG. 3 is a timing diagram illustrating training and prediction timeintervals, according to some embodiments of the present disclosure;

FIG. 4 is a flowchart of recurrent neural network training operationsaccording to various embodiments of the present disclosure; and

FIG. 5 is a flowchart of tumor movement prediction operations accordingto various embodiments of the present disclosure.

Although the following Detailed Description will proceed with referencebeing made to illustrative embodiments, many alternatives,modifications, and variations thereof will be apparent to those skilledin the art.

DETAILED DESCRIPTION

A recurrent neural network (RNN) is a class of artificial neural networkwhere present network outputs may be influenced by past events. RNNs maythus be configured to model dynamics of time varying data. An RNN mayinclude, but is not limited to, a reservoir computing (RC) network. AnRC network may include an input layer, a hidden layer, and output layer.Each layer may include one or more neurons or nodes. The RC hidden layermay correspond to a “reservoir” of neurons in fixed connections that maybe random. The neurons in the reservoir may be recurrently, sparsely,statically, and/or randomly connected. In one nonlimiting example, thereservoir may correspond to a time delay reservoir. A time delayreservoir may correspond to a delay line topology with a number (e.g.,on the order of 100) of neurons stored in a temporal domain as virtualnodes along the delay line. In another example, the reservoir maycorrespond to a true-time delay reservoir. Advantageously, during RCnetwork training, only weights connecting the hidden layer to outputlayer neurons are updated. This simplicity in network topology andtraining may provide speed, size, weight, and power (SWaP) efficiencies.

An RC network may be implemented in circuitry (that may beprogrammable), firmware and/or software and, in some situations, atleast a portion of the RC network may be implemented in the opticaldomain. In one embodiment, the RC network may include a photonictrue-time delay reservoir (TDR). As used herein, “true time delay” meansthat a time delay of the reservoir corresponds to intrinsic time delaysof the electronic components and the optical components included in thereservoir rather than a time delay introduced into the reservoir by afiber spool. A true-time delay may correspond to a characteristic of thesystem and may correspond to a minimum time delay. In anotherembodiment, the RC network may include a fiber optic spool configured toprovide delay so that delays include not only intrinsic delays ofelectronics and coupling fiber but also delay provided by the fiberspool. The RC network, including the time delay reservoir (or true-timedelay reservoir) may be configured for, for example, real-timerespiratory motion prediction.

Generally, during training, an RC network is configured to learn adesired expected value of an l-dimensional output y(k) as a function ofan n-dimensional input u(k) , given data sequence(s) of length k=1, 2 .. . , K. The n-dimensional input u(k) may then correspond to a sampledversion of a continuous input signal u(t), with a sample period, τ. Thereservoir may correspond to an N-dimensional space x (k) and the inputu(k) may be projected into the reservoir. N may be greater than K. For aselected discrete time input u(k), the reservoir state may then bewritten as:

x(k)=f(βW _(in) u(k)+αW _(res) x(k−1)+φ)   (1)

where f is a nonlinear transfer function of the reservoir, β is inputgain, α is attenuation of the reservoir state, φ corresponds to a biasvector, W_(in) is an n×N matrix of random projection weights between theinput layer neurons and the reservoir, and W_(res) is an N×N matrixdescribing the internal connections and individual weights of thereservoir neurons. Both α and β may be on the interval [0,1]. res

The RC response ŷ(k) (e.g., predicted future position) may then be alinear sum of the weighted reservoir output neurons:

ŷ(k)=W _(out) S(k),   (2)

where S(k) is an (N+1)×K matrix concatenating the reservoir states x(k)with a fixed output bias neuron of value 1 and W_(out) is an l×(N+1)matrix of trained output weights. Using the target output y(k) for thetraining examples, W_(out) may be calculated directly by minimizing aMean Squared Error (MSE) between the target output y(k) and thereservoir output via, for example, a regression as:

W _(out) =yS ^(T)(SS ^(T) +λI)⁻¹   (3)

where T denotes transpose, λ is a regularization factor, and I is anidentity matrix. It may be appreciated that the most computationallyexpensive operation is the initial matrix inverse calculation.

Generally, this disclosure relates to a system for tumor movementprediction. A tumor may be included in or may correspond to a targetregion in a patient. In an embodiment, a system may be configured tocapture target region position data and to predict, in real time, afuture position of the target region. The future position informationmay then be used to facilitate imaging and/or provision of therapeuticenergy to the target region. In one nonlimiting example, the targetregion may correspond to a tumor that is to be irradiated. The positionof the target region may vary over the imaging/therapeutic session due,in part, to patient respiration. As is known, adult respiration, atrest, may typically be the range of 12 to 20 breaths per minute. Therespiration of infants and young children is generally higher. Adultrespiration may similarly be higher in some situations. Thus, althoughgenerally periodic, respiration may vary.

In an embodiment, the predicting may be performed by a recurrent neuralnetwork (RNN) circuitry that has been trained, in real time, usingpatient target region position data. The training may be updated using asliding window of position data, to reflect relatively more currentpatient respiration and corresponding target region position data. Asused herein, “real time” means initial training of the recurrent neuralnetwork circuitry may be performed in less than one minute. In onenonlimiting example, initial training of the recurrent neural networkcircuitry may be performed in less than or equal to 30 seconds.Subsequent updating may be performed during a sampling time interval. Inone nonlimiting example, the sampling time interval may be on the orderof tens of milliseconds. However, this disclosure is not limited in thisregard.

Thus, patient respiration may be accommodated and theimaging/therapeutic devices may continue to accurately target thepatient target region in the presence of patient respiration. Trainingand updating in real time are configured to capture relatively currentrespiration data and to facilitate relatively accurately predictingtarget region position. Thus, imaging or irradiating non-target regionsmay be reduced or avoided.

In an embodiment, there is provided a method of predicting respiratorymotion in real-time. The method includes training, by a training module,a recurrent neural network circuitry in real time. The training isperformed over a training time interval. The training is based, at leastin part, on a training data set. The training data set includes atraining number of measured position samples. The measured positionsamples are related to respiratory motion of a patient target region.The method further includes predicting, by the trained recurrent neuralnetwork circuitry, a future position of the patient target region at afuture point in time based, at least in part, on a prediction data set.The future point in time is a look ahead time interval in the futurerelative to a prediction time interval. The prediction data set includesa prediction number of measured position samples.

FIG. 1 illustrates a functional block diagram of a system 100 for tumormovement prediction, consistent with several embodiments of the presentdisclosure. System 100 includes a recurrent neural network (RNN)circuitry 102 and a system controller circuitry 104. System 100 mayinclude or may be coupled to a position sensing circuitry 106. Positionsensing circuitry 106 is configured to detect motion of target region112 that may be due to or related to respiration of a test subject 110.System 100 may be coupled to an imaging/therapeutic device 108.Imaging/therapeutic device 108 is configured to image and/or providetherapeutic radiation to the test subject 110 and, in particular, to atarget region 112. Imaging/therapeutic device 108 may include, but isnot limited to an x-ray scanner, a computed tomography (CT) scanner, amagnetic resonance imaging (MM) scanner, a proton scanner, etc.

Test subject 110 may correspond to a patient and target region 112 maycorrespond to at least a portion of the patient that is to be imagedand/or is to receive therapeutic radiation. In one nonlimiting example,target region 112 may correspond to a tumor that is to be imaged and/orirradiated. It may be appreciated that patient respiration may cause thetarget region 112 to move during imaging and/or provision of therapeuticradiation. Such respiratory motion may result in imaging only a portionof the target region 112 and/or unintentionally irradiating tissueadjacent the target region 112.

Position sensing circuitry 106 may be positioned in or on test subject110 relative to target region 112. Position sensing circuitry 106 may beconfigured to capture the respiratory motion continuously and/ordiscretely, over time, and to provide corresponding measured positiondata 103 to system controller circuitry 104. The measured position data103 corresponds to u(t), as described herein. A measured position samplemay then be u(k) that corresponds to a sampled u(t), sampled with asample period, τ, as described herein. The measured position data 103may thus represent respiratory motion and corresponding motion of thetarget region 112 caused by the respiratory motion.

In one embodiment, recurrent neural network (RNN) circuitry 102 maycorrespond to, for example, an RC network. Generally, RNN 102 isconfigured to receive sampled measured position data 105 (i.e., measuredposition samples) corresponding to measured position data 103 fromsystem controller circuitry 104 and to provide a predicted position 113as output, as will be described in more detail below.

System controller circuitry 104 includes processor circuitry 120,input/output (I/0) circuitry 122, and memory circuitry 124. Systemcontroller circuitry 104 may further include a training module 126 and aprediction module 128. System controller circuitry 104 may furtherinclude mask source circuitry 130, sample and hold circuitry 132, and adata store 136. In some embodiments, system controller circuitry 104 mayinclude input position circuitry 134. In some embodiments, positionsensing circuitry 106 and/or input position circuitry 134 may include aGaussian filter configured to reduce noise that may be present in themeasured position data 103.

The data store 136 may be included, for example, in memory circuitry124, and may be configured to store one or more of position data 136-1,reservoir state data 136-2, output weight data 136-3, and/orconfiguration data 136-4. Position data 136-1 may include one or moresets of measured position data 103 received from position sensingcircuitry 106. Reservoir state data 136-2 may include reservoir statedata 111 captured from recurrent neural network circuitry 102, as willbe described in more detail below. Output weight data 136-3 may includeone or more sets of output weight data (e.g., output weight matrix,W_(out)) determined based, at least in part, on measured position data,as will be described in more detail below. Configuration data 136-4 mayinclude, for example, type and characteristic data associated withimaging/therapeutic device 108. Such characteristic data may then beutilized to determine an appropriate future time for a prediction oftarget region position.

System controller circuitry 104 may generally be implemented incircuitry, and may further include software and/or firmware. In anembodiment, system controller circuitry 104 may be implemented inprogrammable circuitry including, for example, a field-programmable gatearray (FPGA).

In operation, system controller circuitry 104 may be coupled torecurrent neural network circuitry 102 and may be configured to manageoperation of RNN 102. System controller circuitry 104 may be configuredto provide data to and receive data from recurrent neural networkcircuitry 102. System controller circuitry 104 may be configured toreceive measured position data 103 from position sensing circuitry 106.The measured position data 103 may then be stored in data store 136 asposition data 136-1. System controller circuitry 104 may then beconfigured to provide sampled measured position data 105 and mask data107 to recurrent neural network circuitry 102, as will be described inmore detail below. System controller circuitry 104 may be configured toreceive predicted position data 113 from recurrent neural networkcircuitry 102 and to provide a predicted future position 115 toimaging/therapeutic device 108. System controller circuitry 104 may befurther configured to provide output weight data 109 to, and to capturereservoir state data 111 from, recurrent neural network circuitry 102.

Training module 126 is configured to manage training recurrent neuralnetwork circuitry 102, as will be described in more detail below.Prediction module 128 may be configured to provide a predicted futureposition 115 of target region 112 to imaging/therapeutic device 108. Thepredicted future position 115 may correspond to predicted position data113 received from recurrent neural network circuitry 102, as will bedescribed in more detail below.

Training module 126 may be further configured to manage updating thetraining of the recurrent neural network circuitry 102, in real time, atan update time interval. The updating may be based, at least in part, ona current internal state of the recurrent neural network, i.e.,reservoir state data 111. In an embodiment, the update time interval maycorrespond to a sample time interval associated with measured positionssamples.

Computationally, operations of system 100 may be described as follows.Continuous measured position data 103 may correspond to an input signalu(t). Thus, a continuous time input signal u(t) may be sampled at asample rate yielding a corresponding discrete sequence, u(k), thatincludes a sequence of measured position samples. The sample rate maycorrespond to position data sample rate, as described herein. A sampletime interval may thus have time duration, r. Sample and hold circuitry132 may be configured to sample and hold the input signal with a sampleand hold period of length, τ. Mask source circuitry 130 may then beconfigured to generate a mask function m(t) that may be based, at leastin part, on a random distribution. The mask function may then be sampledto provide W_(in), a matrix of random projection weights, as describedherein. In one embodiment, W_(in) may be provided to RNN 102 as maskdata 107. In another embodiment, input position circuitry 134 may beconfigured to combine the mask data with the sampled measured positiondata 105 and to provide the result to RNN 102. The sampled measuredposition data 105 and mask data 107 may then be coupled to the RNN via anonlinear node, as described herein.

It may be appreciated that the RNN 102 may include a reservoir and thereservoir is a dynamic system that can be described by a delayeddifferential equation, e.g., Eq. (1), that includes a nonlinearfunction, ƒ. In one nonlimiting example, ƒ may correspond to a sin²function.

The evolution equation of the reservoir states x(k) of the RNN 102 maycorrespond to Eq. (1). The reservoir state data 111 may thus correspondto reservoir states x(k).

Training module 126 may be configured to determine the output weightmatrix W_(out) (i.e., output weight data 109) based, at least in part,on reservoir state data 111 that corresponds to x(k). It may beappreciated that determining the output weight matrix corresponds totraining RNN 102. Initially, training may be performed for a trainingtime interval and may be based, at least in part, on a training dataset. The training data set may include a training number of measuredposition samples. In one nonlimiting example, the training number ofmeasured position samples may be on the order of 100, e.g., 500.However, this disclosure is not limited in this regard. A duration ofthe training time interval and a size of the training data set may berelated to a prediction accuracy. In one nonlimiting example, thetraining time interval may be less than 30 seconds. However, thisdisclosure is not limited in this regard. The training number ofmeasured position samples may be related to the training time intervalby a position data sample rate. In some embodiments, the position datasample rate may be related to the sample and hold period associated withthe RNN.

The output weight matrix may be determined via, for example, aregression. The regression, may include, but is not limited to, a ridgeregression or a linear regression. Thus, W_(out) may be determined byminimizing a mean squared error (MSE) between a target output y(t) and apredicted ŷ(t), (Eq. (3)). As used herein, a target (i.e., measured)output y(t) corresponds to a future position of the patient targetregion at a future point in time that is a look ahead time interval inthe future relative to a prediction time interval. The target output maybe included in a training data set that may be stored in position data136-1, for example. The output weight matrix W_(out) (i.e., outputweight data 109) may then be provided to RNN 102. The predicted positiondata 113 may then correspond to an output of RNN 102, i.e., ŷ(k), alinear sum of the weighted reservoir output neurons (Eq. (2)), weightedby the output weight data 109. Thus, system 100 (i.e., RNN 102 andsystem controller circuitry 104) may be configured to receive measuredposition data corresponding to motion of target region 112 related torespiration. The measured position data may be sampled and held to yieldsampled measured position data and a mask function may be configured toprovide mask data. The sampled measured position data 105 and the maskdata 107 may then be provided to RNN 102. Training includes determiningthe output weight matrix based, at least in part, on the input positiondata, u(t), and based, at least in part on target measured positiondata. The target measured position data, y(t), corresponds to a time inthe future relative to the input position data, u(t).

After training, i.e., after determining the output weight data 109(W_(out)), prediction module 128 may be configured to utilize RNN 102 topredict a future position of the patient target region at a future pointin time. The prediction may be based, at least in part, on a predictiondata set. The prediction data set may include a prediction number ofmeasured position samples that may be captured and stored in positiondata 136-1. In one nonlimiting example, the prediction number ofmeasured position samples may be on the order of 10, e.g., may be 15.However, this disclosure is not limited in this regard. The future pointin time may correspond to a look ahead time interval in the futurerelative to a prediction time interval, as described herein. Thus, thetrained RNN 102 may be configured to predict the future position of thepatient target region at the future point in time based, at least inpart, on the prediction data set.

In an embodiment, the output data matrix, W_(out), may be updated asmeasured position data is acquired during an imaging/treatment session.In other words, after initial training that determines an initial outputdata matrix, W_(out), as new measured position data is captured, W_(out)may be updated and the updated output data matrix may then be used bythe RNN to predict the future position.

FIG. 2 illustrates a functional block diagram of an example recurrentneural network (RNN) 200 consistent with several embodiments of thepresent disclosure. RNN 200 is one example of recurrent neural networkcircuitry 102 of FIG. 1. RNN 200 includes an input light source 202,input modulation circuitry 204, a reservoir 208, output modulationcircuitry 210, and output conversion circuitry 212. For example,reservoir 208 may correspond to an optical reservoir and RNN 200 maythen correspond to an optical reservoir computing device. Reservoir 208includes a combiner circuitry 220, nonlinear node circuitry 222, anddelay optical circuitry 224. Delay optical circuitry 224 includes aplurality of virtual nodes 224-0, 224-1, . . . , 224-N-1. Input lightsource 202 and input modulation circuitry 204 may correspond to an inputlayer. Reservoir 208 may correspond to a reservoir layer. Outputmodulation circuitry 210 and output conversion circuitry 212 maycorrespond to an output layer of RNN 200.

A reservoir, e.g., reservoir 208, may include one or more nonlinearneurons. In an embodiment, reservoir 208 may correspond to a time delayreservoir (TDR) that may be configured as a ring topology that includesone nonlinear neuron (e.g., nonlinear node circuitry 222) and anarbitrary number N“virtual nodes”, e.g., virtual nodes 224-0, 224-1, . .. , 224-N-1. The ring may then correspond to a delay line, e.g.,including combiner circuitry 220, nonlinear node circuitry 222, anddelay optical circuitry 224. The TDR may be configured to distributeinformation in time across the plurality of virtual nodes 224-0, 224-1,. . . , 224-N-1 in the delay line 224.

In an embodiment, reservoir 208 may correspond to a true-time delayreservoir, as described herein. In another embodiment, reservoir 208(e.g., delay optical circuitry 224) may include a fiber optic spoolconfigured to provide delay so that delays include not only intrinsicdelays of electronics and coupling fiber but also delay provided by thefiber spool. In this embodiment, reservoir 208 may then correspond to atime delay reservoir.

In an embodiment, the input signal to the reservoir 208 may be sampledand held for time τ. Time duration r may correspond to a duration of oneround-trip through the delay line reservoir, e.g., reservoir 208. Then,for each sampled input u(k), the delay line with characteristic durationτ may itself be sampled N+1 times with sample period θ, where τ=(N+1)θ.Each measurement, i.e., each delay line sample, may correspond to avirtual node, e.g., virtual nodes 224-0, 224-1, . . . , 224-N−1. As usedherein, a virtual node may not have a defined physical location and maynot be configured to perform computation on the signal. It may beappreciated that, while the reservoir may be configured such that N+1virtual nodes are supported, typically only N of those virtual nodes areused for determining the reservoir state, x(k).

It may be appreciated that merely sampling u(k) over τ would result inall virtual nodes, e.g., virtual nodes 224-0, 224-1, . . . , 224-N−1,having a same value. Thus, mask source circuitry 130 may be configuredto provide a masking function of fixed arbitrary weights (W_(in)) thatmay then be applied to each u(k), as described herein. The result may belinearized and then sequentially fed to the reservoir, i.e., to combinercircuitry 220. In one nonlimiting example, the masking function maycorrespond to a uniform random mask with range [−1, +1]. However, thisdisclosure is not limited in this regard.

Input modulation circuitry 204 may be configured to receive light energyfrom input light source 202 and to modulate the received light based, atleast in part, on sampled measured position data 105 and mask data 107.In some embodiments, input light source 202 may correspond to a laser.In some embodiments, input light source 202 may correspond to alight-emitting diode. In one nonlimiting example, input modulationcircuitry 204 may correspond to a Mach Zehnder modulator. However, thisdisclosure is not limited in this regard.

The modulated output of input modulation circuitry 204 may then beprovided to reservoir 208 and the combiner circuitry 220. Combinercircuitry 220 may be further configured to receive an output of delayoptical circuitry 224 and virtual node N−1 224-N−1. An output ofcombiner circuitry 220 may then be provided to nonlinear node circuitry222. In one nonlimiting example, nonlinear node circuitry 222 may beconfigured to implement a sin² function. However, this disclosure is notlimited in this regard. An output of nonlinear node circuitry 222 maythen be provided to delay optical circuitry 224 and virtual node 0224-0. An output of virtual node 0 224-0 may then be provided to virtualnode 1 224-1 and so on to virtual node N−1 224-N−1. A state of eachvirtual node may correspond to reservoir state data 111 that may becaptured and provided to system controller circuitry 104, as describedherein. The reservoir state data 111 may also be provided to outputmodulation circuitry 210. Output modulation circuitry 210 may be furtherconfigured to receive output weight data 109. For example, output weightdata 109 may be received from system controller circuitry 104. Outputmodulation circuitry 210 may then be configured to modulate thereservoir state data 111 with the output weight data 109. It may beappreciated that output weight data 109 corresponds to an output weightmatrix. An output of output modulation circuitry 210 may then beprovided to output conversion circuitry 212. Output conversion circuitry212 is configured to convert the output of output modulation circuitry210 into predicted position data 113. In one nonlimiting example, outputconversion circuitry 212 may be configured to convert optical data intoelectrical data.

Thus, RNN 200 may be configured to receive sampled measured positiondata 105 and mask data 107 and to provide, as output, predicted positiondata 113.

It may be appreciated that prediction using neural networks generallyincludes training the neural network and then predicting using thetrained neural network. Generally, training a neural network isrelatively more computationally expensive compared to predictionoperations using the neural network. A measured position data set maygenerally include a sequence of samples of measured position,corresponding to respiration motion. An amount of samples is related toboth the sample rate and a total time duration of the data set. Forexample, one minute of measured position data captured at a sample rateof 33 samples per second yields 33 samples/sec×60 sec/minute×1minute=1980 samples, or approximately 2000 samples. To manage trainingcomplexity, an initial portion (e.g., training data set) of the measuredposition data set may be used for training to determine an optimized butfixed W_(out). The determined W_(out) may then be used for motionprediction operations based, at least in part, on a remaining portion ofthe measured position data. In one nonlimiting example, a measuredposition data set of duration 2 minutes may be divided into an initialportion of duration 90 seconds (˜3000 data points at 33 samples/sec) fortraining and the remaining 30 seconds for prediction operations.

It may be appreciated that a relatively longer training period withassociated greater number of measured position samples may result in arelatively better prediction accuracy. For example, the longer durationtraining data may include more breathing pattern variations so that theRNN may have more “memories” of any irregularities. Clinically, a longerdata training session may result in an extended radiation treatment timeand thus reduced efficiency. Additionally or alternatively, an extendedtraining data set may result in over-fitting of the RNN model for somepatients' breathing patterns. Additionally or alternatively, a patient'sbreathing pattern may change over time so that an earlier trained RNNmay become less accurate as a prediction time becomes relatively moretemporally distant from the training time interval.

FIG. 3 is a timing diagram 300 illustrating training and prediction timeintervals, according to some embodiments of the present disclosure.Timing diagram 300 includes a plot 302 with a horizontal axiscorresponding to time in seconds and a vertical axis corresponding toposition in millimeters. Timing diagram 300 further includes a measuredposition waveform 304 corresponding to patient respiration. Timingdiagram 300 includes a training time interval 310 and a predictionregion k_(p) 320. Timing diagram 300 further includes a plurality oftiming blocks 312-1, 312-2, . . . , 312-m and 312-p configured toillustrate time intervals related to training and prediction, as will bedescribed in more detail below. The timing blocks are grouped into atraining group 306 and a prediction 308. Training group 306 may thusinclude timing blocks 312-1, 312-2, . . . , 312-m and prediction 308includes timing block 312-p. Each timing block 312-1, 312-2, . . . ,312-m and 312-p includes a window of length k_(w), that corresponds to aprediction time interval 314; a look ahead time interval 316 withlength, h; and a prediction point (i.e., k_(out)) 318. The windowlength, k_(w), may thus correspond to a prediction number of measuredposition samples. Thus, a length of each timing block may correspond toa number of measured position samples (i.e., data points). Theprediction time interval 314 corresponds to a number of measuredposition samples to be used for a prediction. The look ahead timeinterval 316 corresponds to a number of sample periods between theprediction time interval 314 (i.e., an end of the prediction timeinterval 314) and the prediction 318.

In operation, the output weight matrix W_(out) may be determined duringthe training time interval 310. A length, k_(tr), of the training timeinterval 310 corresponds to a number of measured position data points(i.e., samples) used for training the RNN. It may be appreciated thatthe number of measured data points used for training affects predictionperformance. The length, k_(tr), of the training time interval 310 and,thus, the number of measured data points used for training are relatedto a time duration of the training time interval by the position datasample rate. In other words, the time duration of the training timeinterval is equal to k divided by the position data sample rate. In anembodiment, the number of measured data points, k_(tr), may be in therange of 300 to 1000. In one embodiment, k_(tr) may be 500. At aposition data sample rate of 33 samples/sec, k_(tr)=1000 corresponds totime duration of the training time interval of about 30 seconds. In onenonlimiting example, k may be 500 data points (corresponding to a timeduration of the training time interval of about 15 seconds at positiondata sample rate of 33 samples/sec).

The length of the training time interval k_(tr) may be fixed at aselected value (e.g., 500) and may then correspond to a sliding window.In other words, the training may proceed based, at least in part, on thetraining time interval and the training time interval may be measuredfrom a current time back in time for an interval duration defined byk_(tr). The training time interval may be configured to slide throughsome or all of an imaging/therapeutic session. Thus, the training dataset may correspond to a sliding window. In other words, with each newmeasured position sample added to the training data set, an oldestmeasured position sample may be dropped from the training data set. Thetraining data set may be updated at the measured position data samplingrate, e.g., 33 samples/sec.

The output weight matrix W_(out) may then be updated at approximatelythe measured position data sampling rate, e.g., each 30 milliseconds fora sample rate of 33 samples/second. The output weight matrix may thus beconfigured to reflect a most recent breathing pattern and may be updatedthroughout the imaging/therapeutic session. It may be appreciated that arelatively smaller k_(tr) may correspond to a relatively bettercomputational efficiency, i.e., a relatively shorter time to determine,e.g., W_(out).

Each timing block 312-1, 312-2, . . . , 312-m and 312-p includes aprediction time interval (i.e., window length, k_(w)) 314, a look aheadtime interval 316 with length, h, and a prediction point (i.e., k_(m))318. The prediction time interval 314 corresponds to a number ofmeasured position samples to be used for a prediction. The look aheadtime interval 316 corresponds to a number of sample periods between theprediction time interval 314 (i.e., an end of the prediction timeinterval 314) and the prediction 318. A relatively larger k_(w) mayproduce a relatively better prediction results, however, the improvementmay be somewhat asymptotic. In other words, increasing k_(w) beyond aselected number may result in a lower improvement in predictionaccuracy. In an embodiment, k_(w), i.e., a prediction number of measuredposition samples, may be in the range of 5 to 20. In one nonlimitingexample, lc, may be equal to 15.

The look-ahead length, h, may be measured from a last data point in theprediction time interval k_(w). In some situations, h may be related toimaging/therapeutic equipment latency. Imaging/therapeutic equipmentlatency may be specific to a particular imaging/therapeutic device. Forexample, latency may be in the range of 40 ms (milliseconds) to 500 ms,depending on the particular imaging/therapeutic device. In anembodiment, h may be in the range of 1 to 20. For example, for a 30 mssample interval, h equal to 2 corresponds to a look-ahead time of 60 ms.In another example, h equal to 10 corresponds to a look-ahead time of300 ms.

Thus, a system may be configured to capture target region position dataand to predict, in real time, a future position of the target region.The future position information may then be used to facilitate imagingand/or provision of therapeutic energy to the target region. Theposition of the target region may vary over the imaging/therapeuticsession due, in part, to patient respiration. In an embodiment, thepredicting may be performed by a recurrent neural network (RNN)circuitry that has been trained, in real time, using patient targetregion position data. The training may be updated using a sliding windowof position data, to reflect relatively more current patient respirationand corresponding target region position data.

FIG. 4 is a flowchart 400 of recurrent neural network trainingoperations according to various embodiments of the present disclosure.In particular, the flowchart 400 illustrates training a recurrent neuralnetwork based, at least in part, on a training data set. The operationsmay be performed, for example, by system controller circuitry 104 andrecurrent neural network circuitry 102 of FIG. 1.

Operations of this embodiment may begin with receiving measured positiondata at operation 402. Measured position data may be sampled atoperation 404. Mask data may be generated at operation 406. A reservoirstate may be determined at operation 408. Output weight data may bedetermined at operation 410. Operation 412 may include repeatingoperations 402 through 410. Operations may then continue at operation414. Thus, a recurrent neural network may be trained based, at least inpart, on a training data set

FIG. 5 is a flowchart 500 of tumor movement prediction operationsaccording to various embodiments of the present disclosure. Inparticular, the flowchart 500 illustrates predicting a future targetregion position by a recurrent neural network based, at least in part,on measured position data. The operations may be performed, for example,by recurrent neural network 102 and system controller circuitry 104 ofFIG. 1.

Operations of this embodiment may begin with receiving measured positiondata at operation 502. Measured position data may be sampled atoperation 504. Sampled data may be provided to a trained recurrentneural network at operation 506. A reservoir state may be determined atoperation 508. A future position of a patient target region may bedetermined at operation 510. Operation 512 may include providing thepredicted future position to an imaging/therapeutic device. Operationsmay continue at operation 514. Thus, a trained recurrent neural networkmay be configured to predict a future position of a patient targetregion based, at least in part, on a prediction data set.

As used in any embodiment herein, the term “module” may refer to an app,software, firmware and/or circuitry configured to perform any of theaforementioned operations. Software may be embodied as a softwarepackage, code, instructions, instruction sets and/or data recorded onnon-transitory computer readable storage medium. Firmware may beembodied as code, instructions or instruction sets and/or data that arehard-coded (e.g., nonvolatile) in memory devices.

“Circuitry”, as used in any embodiment herein, may include, for example,singly or in any combination, hardwired circuitry, programmablecircuitry such as a computer processor including one or more individualinstruction processing cores, a microcontroller, a field programmablegate array (FPGA), state machine circuitry, and/or firmware that storesinstructions executed by programmable circuitry. Circuitry may includeelectrical circuits and elements, optical fibers and elements and/oroptical/electrical conversion circuitry. A module may, collectively orindividually, be embodied as circuitry that forms part of a largersystem, for example, an integrated circuit (IC), an application-specificintegrated circuit (ASIC), a system on-chip (SoC), desktop computers,laptop computers, tablet computers, servers, smart phones, etc.

Memory circuitry 124 may include one or more of the following types ofmemory: semiconductor firmware memory, programmable memory, non-volatilememory, read only memory, electrically programmable memory, randomaccess memory, flash memory, magnetic disk memory, and/or optical diskmemory. Either additionally or alternatively system memory may includeother and/or later-developed types of computer-readable memory.

Embodiments of the operations described herein may be implemented in acomputer-readable storage device having stored thereon instructions thatwhen executed by one or more processors perform the methods. Theprocessor may include, for example, a processing unit and/orprogrammable circuitry. The storage device may include a machinereadable storage device including any type of tangible, non-transitorystorage device, for example, any type of disk including floppy disks,optical disks, compact disk read-only memories (CD-ROMs), compact diskrewritables (CD-RWs), and magneto-optical disks, semiconductor devicessuch as read-only memories (ROMs), random access memories (RAMs) such asdynamic and static RAMs, erasable programmable read-only memories(EPROMs), electrically erasable programmable read-only memories(EEPROMs), flash memories, magnetic or optical cards, or any type ofstorage devices suitable for storing electronic instructions.

The terms and expressions which have been employed herein are used asterms of description and not of limitation, and there is no intention,in the use of such terms and expressions, of excluding any equivalentsof the features shown and described (or portions thereof), and it isrecognized that various modifications are possible within the scope ofthe claims. Accordingly, the claims are intended to cover all suchequivalents.

Various features, aspects, and embodiments have been described herein.The features, aspects, and embodiments are susceptible to combinationwith one another as well as to variation and modification, as will beunderstood by those having skill in the art. The present disclosureshould, therefore, be considered to encompass such combinations,variations, and modifications.

What is claimed is:
 1. A method of predicting respiratory motion in realtime, the method comprising: training, by a training module, a recurrentneural network circuitry in real time, the training performed over atraining time interval, the training based, at least in part, on atraining data set, the training data set comprising a training number ofmeasured position samples, the measured position samples related torespiratory motion of a patient target region; predicting, by thetrained recurrent neural network circuitry, a future position of thepatient target region at a future point in time based, at least in part,on a prediction data set, the future point in time a look ahead timeinterval in the future relative to a prediction time interval, theprediction data set comprising a prediction number of measured positionsamples.
 2. The method of claim 1, further comprising updating, by thetraining module, the training of the recurrent neural network circuitry,at an update time interval, the updating based, at least in part, on acurrent internal state of the recurrent neural network circuitry.
 3. Themethod of claim 1, wherein the training comprises repeatedly determininga plurality of output weight values using a sliding window correspondingto the prediction number of measured position samples.
 4. The method ofclaim 1, wherein the training data set corresponds to a sliding window.5. The method of claim 1, wherein the prediction time interval is lessthan one second.
 6. The method of claim 2, wherein the update timeinterval is related to a position data sample rate.
 7. The method ofclaim 1, wherein the recurrent neural network corresponds to a reservoircomputing network.
 8. The method of claim 1, wherein the training timeinterval is less than or equal to 30 seconds.
 9. The method of claim 1,wherein the prediction number of samples is in the range of 5 to
 20. 10.The method of claim 6, wherein the position data sample rate is in therange of 30 to 40 samples per second.
 11. A system for predictingrespiratory motion in real time, the system comprising: a systemcontroller circuitry; a training module; and a recurrent neural networkcircuitry, wherein the training module is configured to train therecurrent neural network circuitry in real time, the training isperformed over a training time interval, the training is based, at leastin part, on a training data set, the training data set comprises atraining number of measured position samples, and the measured positionsamples are related to respiratory motion of a patient target region;and the trained recurrent neural network circuitry is configured topredict a future position of the patient target region at a future pointin time based, at least in part, on a prediction data set, the futurepoint in time is a look ahead time interval in the future relative to aprediction time interval, and the prediction data set comprises aprediction number of measured position samples.
 12. The system of claim11, wherein the training module is configured to update the training ofthe recurrent neural network circuitry, at an update time interval, theupdating based, at least in part, on a current internal state of therecurrent neural network circuitry.
 13. The system of claim 11, whereinthe training comprises repeatedly determining a plurality of outputweight values using a sliding window corresponding to the predictionnumber of measured position samples.
 14. The system of claim 11, whereinthe training data set corresponds to a sliding window.
 15. The system ofclaim 11, wherein the prediction time interval is less than one second.16. The system of claim 12, wherein the update time interval is relatedto a position data sample rate.
 17. The system of claim 11, wherein therecurrent neural network corresponds to a reservoir computing network.18. The system of claim 11, wherein the training time interval is lessthan or equal to 30 seconds.
 19. The system of claim 11, wherein theprediction number of samples is in the range of 5 to
 20. 20. The systemof claim 16, wherein the position data sample rate is in the range of 30to 40 samples per second.